﻿ Activity Recognition and Monitoring Using Multiple Sensors on Different Body Positions Uwe Maurer1, Asim Smailagic2, Daniel P Siewiorek2, Michael Deisher3, 2 School of Computer Science, Carnegie Mellon University, Pittsburgh 1 Computer Science Department, Technische Universität München, Germany 3 Intel, Hillsboro, OR Abstract The design of an activity recognition and monitoring sys- tem based on the eWatch, multi-sensor platform worn on different body positions, is presented in this paper The sys- tem identiﬁes the user’s activity in realtime using multiple sensors and records the classiﬁcation results during a day We compare multiple time domain feature sets and sam- pling rates, and analyze the tradeoff between recognition accuracy and computational complexity The classiﬁcation accuracy on different body positions used for wearing elec- tronic devices was evaluated Figure 1 eWatch sensing platform sign method for a context recognition system is proposed 1 IntroductionIt evaluates multiple feature sets and makes the tradeoff be- The primary goal of this paper is to study the effective-tween power consumption and recognition accuracy A sys- ness of activity classiers in a multi-sensor system as wetem that classies household activities in realtime with a vary the wearing positions of the sensors The eWatch isfocus on low power consumption is presented in used as a multi sensor platform for wearable context-awareIn , a system using an armband based sensor array computing Previous feature extraction studies examiningand unsupervised machine learning algorithms was able to accelerometer data have shown that it is a viable input fordetermine a meaningful user context model detecting user states when it is worn on the wrist Moti-In Section 2, we describe the sensor platform and Sec- vated by other possible sensor platform locations, especiallytion 3 explains the experimental design Section 4 describes with mobile communication devices such as a cell phone orthe activity recognition method Section 5 presents the re- PDA, we designed a study to investigate the dependency ofsults of the data analysis, and Section 6 addresses the per- the eWatch classication accuracy on given different bodyformance of our on-board activity classier positions We investigate wearing the eWatch in the follow- 2 Sensing Platform ing locations: the belt, shirt pocket, trouser pocket, back- pack, and necklace The results of the study would help usOur sensor platform, the eWatch (Figure 1), is based on decide on the best position to place such a sensor platform,the Philips LPC2106 ARM7 TDMI microcontroller, with and understand the nature of the trade-off between wearing128kB of internal ash memory and 64kB of RAM position and classication performance The LPC2106 is a 32bit processor running at up to 60Mhz In , the authors used multiple accelerometers worn oneWatch contains four sensors: a dual axes accelerometer, a person’s body to recognize their physical activity Sensorlight, temperature sensor and microphone Sensor data can data from multiple body positions was combined for classi-be stored in a 1MB external ash memory fying the activities In , a low power sensor hardware system is pre-3 Experiment Design sented, including accelerometer, light sensor, microphone,In our study we focussed on six primary activities:sit- and wireless communication Based on this hardware, a de-ting, standing, walking, ascending stairs, descending stairs Proceedings of the International Workshop on Wearable and Implantable Body Sensor Networks (BSN’06) 0-7695-2547-4/06 $20 00 © 2006 IEEE Report Documentation PageForm No 0704-0188 ApprovedOMB Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302 Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number 1 REPORT DATE 3 DATES COVERED APR 2006 2 REPORT TYPE 00-00-2006 to 00-00-2006 4 TITLE AND SUBTITLE 5a CONTRACT NUMBER Activity Recognition and Monitoring Using Multiple Sensors on Different 5b GRANT NUMBER Body Positions 5c PROGRAM ELEMENT NUMBER 6 AUTHOR(S) 5d PROJECT NUMBER 5e TASK NUMBER 5f WORK UNIT NUMBER 7 PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8 PERFORMING ORGANIZATION Carnegie Mellon University,School of Computer Science,5000 Forbes REPORT NUMBER Ave,Pittsburgh,PA,15213 9 SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10 SPONSOR/MONITOR’S ACRONYM(S) 11 SPONSOR/MONITOR’S REPORT NUMBER(S) 12 DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited 13 SUPPLEMENTARY NOTES 14 ABSTRACT The design of an activity recognition and monitoring system based on the eWatch, multi-sensor platform worn on different body positions, is presented in this paper The system identifies the user?s activity in realtime using multiple sensors and records the classification results during a day We compare multiple time domain feature sets and sampling rates, and analyze the tradeoff between recognition accuracy and computational complexity The classification accuracy on different body positions used for wearing electronic devices was evaluated 15 SUBJECT TERMS 16 SECURITY CLASSIFICATION OF: 17 LIMITATION OF 18 NUMBER19a NAME OF ABSTRACT OF PAGES RESPONSIBLE PERSON a REPORT b ABSTRACT c THIS PAGE Same as4 unclassified unclassified unclassified Report (SAR) Standard Form 298 (Rev 8-98) Prescribed by ANSI Std Z39-18 andrunning Body positions that are normally used for wearing electronic devices, such as cell phones or PDAs, were studied We placed our sensor hardware on the left wrist, belt, necklace, in the right trouser pocket, shirt04 pocket, and bag The subjects wore six eWatch devices lo- 02 cated at these body positions during the study The devices recorded sensor data from the accelerometer and light sen-0 sor into their ash memory The user was asked to perform02 tasks that consist of the activities, such as working on the 04 computer or walking to another building The lead experi- menter annotated the current activity and instructed the sub-06 running jects on how to proceed The annotations were done using08 walking an application running on an extra eWatch worn by the lead0 1standing experimenter sitting descending Six subjects participated in the study, each subject per-0 1ascending 0 3 formed the given tasks in 45 to 50 minutes In total we 0 20 05 collected over 290 minutes of sensor data 0 1 0 0 Sensor setupeWatch recorded both axes of the ac- celerometer and the light sensor All sensors values wereFigure 2 Feature space after LDA transfor- recorded with a frequency of 50Hz and with 8bit resolution mation The accelerometer was calibrated so that both axes operate in a range of±2g Evaluation of the recorded data was done with Matlab and the WEKA software standing, sitting and running activities form seperate clus- ters, whilewalking,ascendinganddescending stairsare 4 Activity recognition methodcloser together since these activities are very similar The sensor values recorded from the accelerometers and the light sensor are split into short time windows TheseFeature SubsetsTo reduce the time and energy required windows are then transformed into the feature space by cal-to calculate the feature vector, several subsets of the com- culating several feature functions over the individual win-plete feature space were evaluated Some features are irrele- dows vant or redundant and do not provide information to signi- cantly improve the classication accuracy Therefore a sub- FeaturesFeatures from both accelerometer axes (X & Y),set of the available features can be selected to decrease the the light sensor, and a combined value of both accelerom-computation time without signicantly decreasing recogni- eter signals were calculated To reduce the dependency ontion accuracy the orientation, bothX and Yvalues were combined calcu-The Correlation based Feature Selection (CFS) method lating the squared length of the acceleration vector Thefrom the WEKA toolkit was used to nd feature sets con- classication accuracy with individual sensors as well as / FunctionNameAvg CPUAvg Time with multiple sensor combined was investigated Features Cyclesinμs Only time domain features were considered to avoid the Empirical Meanmean85414 5 costly computation that is required to transform the signalRoot Mean Squarerm s121920 7 into the frequency domain Table 1 shows the list of fea-Standard Deviationstd113919 3 tures that were considered The functions to calculate theseVariancevar131322 3 Absolute Deviationmad108918 5 features were implemented on the eWatch and the requiredMean Cumulative Histogram (256 bins)hist584799 1 number of clock cycles per function was measured Eachn’ th Percentile(n =5,10, ,95)prc1422 4 function was executed 2000 times with different recordedInterquartile Rangeiqr2894 9 sensor inputs, and then the average value was computed Zero Crossing Ratezcr99316 8 The execution time was calculated based on the measuredMean Crossing Ratemcr99616 9 22 y)131822 3 clock cycles and the CPU frequency at 59MHz Table 1Sq Length of X,Y (x+ Tree classier (18 nodes)1382 3 shows the measured clock cycles and execution time usingDecision a four second window sampled at 20Hz (80 samples) Figure 2 depicts the feature space after a transformationTable 1 List of time domain features and the with Linear Discriminant Analysis (LDA) It shows that theaverage clock cycles and time to calculate them on the eWatch running at 59MHz Proceedings of the International Workshop on Wearable and Implantable Body Sensor Networks (BSN’06) 0-7695-2547-4/06 $20 00 © 2006 IEEE 95959595 wrist 909090pocket90 858585bag85 necklace 80808080 shirt 757575belt75 70707070 65656565 wrist 60wrist606060pocket pocketwrist 5555pocket55Classification accuracy (%)55bag Classification accuracy (%)bagClassification accuracy (%)Classification accuracy (%) bagnecklace 50necklace505050shirt shirtnecklace 4545shirt4545belt belt belt 40404040 051015202530051015202530051015202530051015202530 Sample frequency (Hz)Sample frequency (Hz)Sample frequency (Hz)Sample frequency (Hz) (a) Features from the accelerometer’s(b) Features from the accelerome-(c) Features from the light sensor(d) Features from thex2+y2value X-axister’s Y-Axisof the accelerometers Figure 3 Recognition accuracy with different feature sets taining features that are highly correlated within the particu-wrist 100 lar class but are uncorrelated with each other Table 2 showspocket 90bag the feature sets that were compared necklace 80shirt belt 70 60 Classiﬁcaton methodWe evaluated and compared sev-50 eral classication methods, namely Decision Trees (C4 5 al-40Classification accuracy (%) gorithm), k-Nearest Neighbor (k-NN), Naive-Bayes and the30 descendingrunningsittingstandingascendingwalking Bayes Net classier Decision Trees and Naive-Bayes wereActivity found to achieve high recognition accuracy with acceptable computational complexity Decision Trees were used forFigure 4 Recognition accuracy for the activi- activity classication in and It was shown in thatties at different body locations the discretized version of Naive-Bayes can outperform the Decision Tree classier for general classication problems 5 Data Analysis - eWatch Classiﬁcation Per- Finally the Decision Tree classier was chosen as it pro-formance for Various Wearing Positions vides a good balance between accuracy and computational complexity For all further experiments this classier withWe calculated the classication accuracy for every ac- a 5-fold cross validation was used tivity on each of the six different body positions The data from all subjects was combined to train a general classier that is not specic to a person Sampling frequencyDuring the user study the sensorsTable 2 shows the feature sets and the classication re- were sampled with a frequency of 50Hz and later down-sults for the different body positions The features are cal- sampled to lower frequencies To maintain some of the highculated from a 20Hz signal frequency components information and to reduce the com-Figure 4 shows the recognition accuracy for the individ- putational complexity signicantly, no low pass lter wasual activities at different body locations For the classica- used for downsampling the data Figure 3 shows the recog-tion the reduced feature set F6 was used The data indicate nition accuracy for different sample rates from 1 to 30Hzthat any of the six positions are good for detectingwalking, for the different body positions The recognition accuracystanding, sitting and running Ascending and descending was dened as the percentage of correctly classied featurethe stairs is difcult to distinguish fromwalking in all posi- vectors averaged for all six activities The recognition ac-tions, since the classier was trained for multiple persons curacy increases with higher sampling rates, and with theThe wrist performs best because the feature set was opti- accelerometer features the accuracy then stabilizes betweenmized for the wrist position 15 to 20Hz, and is only improved marginally with higher sampling rates The accuracy with the light sensor only is6 Onboard Activity Classiﬁer lower and it stabilizes beginning with 7Hz In Figure 3(c)Based on these results we implemented a decision tree the results from the belt and pocket position is not shownclassier that runs on the eWatch The feature set F6 was because the light sensor did not provide any useful classi-used to build the decision tree The sensor sampling is cation information at these positions interrupt-based, and triggers the sampling of the sensors at Proceedings of the International Workshop on Wearable and Implantable Body Sensor Networks (BSN’06) 0-7695-2547-4/06 $20 00 © 2006 IEEE #FeaturesCPU CyclesTime inμsClassication Accuracy for Body Position wristpocketbagnecklaceshirtbelt F1All features, all sensors56242953 687 1%85 2%92 8%86 8%89 5%87 0% F2All features from accelerometer X14731249 868 4%78 6%85 2%74 3%76 1%72 6% F3All features from accelerometer Y14731249 883 2%75 4%86 4%61 3%70 1%70 8% F4All features from light14731249 855 0%16 7%18 0%61 7%52 2%48 8% F5All features from accelerometer XY (x2+y2)15049255 276 6%79 5%87 2%72 6%78 0%77 2% F6prc(3),rms,prc(20),prc(97),12114205 487 0%80 1%86 5%78 6%79 6%84 2% yxyyy rms,mad,mean,prc(10) xyy light F8prc(3),iqr,prc(10),prc(97),mad7651129 782 0%62 4%68 9%56 6%69 8%71 7% yyyyx F9rms,qrt,rms,mad,mean10746182 277 3%78 2%80 9%72 3%75 4%76 5% xyxxxyxy Table 2 Feature sub sets and classiﬁcation accuracy for body positions tronic devices walking We will extend our activity classier to other activites ascending and investigate how the activity classication can support standing the recognition of the user’s location sittingActivity 7 1 Acknowledgments running This material is based upon work supported by the De- descending fense Advanced Research Projects Agency (DARPA) under 102030405060708090100 Time in minutesContract No NBCHD030010, the National Science Foun- dation under Grant Nos 0205266 and 0203448, a grant Figure 5 Activity classiﬁcation recorded over from Intel Corporation, and PA Infrastructure (PITA) 100 minutes References 20Hz The sensor value is stored in a buffer with the size of the sliding window The activity is classied every 0 5 sec- L Bao and S S Intille Activity Recognition from User- onds based on the sensor data from the 4 second buffer TheAnnotated Acceleration Data In A Ferscha and F Mattern, Pervasive, volume 3001 ofLecture Notes in Computer classication results are stored into ash memory and areeditors, Science, pages 1–17 Springer, 2004 downloaded to a computer later for further processing and N B Bharatula, M Stäger, P Lukowicz, and G Tröster analysis They can also be transferred in realtime over theEmpirical Study of Design Choices in Multi-Sensor Context Bluetooth connection In order to save energy, the systemRecognition Systems InIFAWC: 2nd International Forum on remains idle between servicing interrupts Applied Wearable Computing, pages 79–93, Mar 2005 A subject wore the eWatch with the built-in activity clas- N B Bharatula, M Stäger, P Lukowicz, and G Tröster wer and Size Optimized Multisensor Context Recognition sier on the wrist during the day The system classied thePo Platform InISWC 2005: Proceedings of the 9th IEEE Inter- activity in realtime and recorded the classication results national Symposium on Wearable Computers, pages 194–195, to ash memory Figure 5 shows 100 minutes of activity Oct 2005 classication, as the user walked to a restaurant, sat down, J Dougherty, R Kohavi, and M Sahami Supervised and ate lunch, went back to the ofce and sat down to con-unsupervised discretization of continuous features InInter- tinue working The classication results match well withnational Conference on Machine Learning, pages 194–202, the actual activities; eating lunch was partially interpreted1995 A Krause, D P Siewiorek, A Smailagic, and J Farringdon aswalking or running activity due to arm movements Unsupervised, dynamic identication of physiological and ac- tivity context in wearable computing InSeventh IEEE In- 7 Conclusions and Future Workternational Symposium on Wearable Computers (ISWC’03), pages 88–97, 2003 The activity recognition and monitoring system that can U Maurer, A Rowe, A Smailagic, and D P Siewiorek identify and record the user’s activity in realtime using mul-eWatch: A Wearable Sensor and Notication Platform In tiple sensors is presented We compared multiple featureBody Sensor Networks Workshop, submitted, 2006 sets and sampling rates to nd an optimized classication I H Witten and E Frank Data Mining: Practical machine method, and showed how well they perform on differentlearning tools and techniques Morgan Kaufmann, San Fran- body locations that are commonly used for wearing elec-cisco, 2nd edition, 2005 Proceedings of the International Workshop on Wearable and Implantable Body Sensor Networks (BSN’06) 0-7695-2547-4/06 $20 00 © 2006 IEEE 